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Preface 



The stated mission of the ACT Office of Policy Research is to inform policy 
makers and the general public on important issues in education by providing 
timely information that can directly enhance knowledge, dialogue, and 
decision making. The current ACT Policy Research Agenda focuses on 
six specific areas: 

■ Developing the Applicant Pool 

■ Increasing Diversity in College 

■ Remedial Education in College 

■ Retention in College 

■ Education and Workforce Transitions 

■ The High School Experience 

ACT policy reports can also be viewed and printed from ACT s website 
(www.act.org/research/policy/index.html). For additional information about 
ACT s policy research work, copies of ACT policy studies, or to contact the 
ACT Office of Policy Research staff, please e-mail us at policy@act.org. 

This study, Evaluating the Effectiveness of Technology in Our Schools , was 
initiated as part of the second authors summer internship at ACT (Boris Volkov 
is a doctoral student in the Educational Policy and Administration Program at 
the University of Minnesota). What began as a discussion of educational 
technology in our nation s schools has resulted in a resource to help 
policymakers and administrators evaluate the effectiveness of technological 
applications implemented to enhance teaching, learning, and achievement. 

This policy report has greatly benefited from the contributions of many 
individuals. Several external-to-ACT educators provided considerable help 
in shaping the study and reviewing draft manuscripts. These individuals 
include James Bosco (Western Michigan University), Douglas Levin 
(American Institutes for Research), and William Mclnerney and Jennifer 
Richardson (Purdue University). The ACT Policy Research Advisory Panel 
provided recommendations about the formulation of the study and reviews 
of draft manuscripts. 

Numerous ACT staff members were involved in various stages of the study. 
The following ACT staff provided help on the structure of the study 
and/or manuscript review: Patricia Farrant, Julie Noble, Wayne Patience, 
Nancy Petersen, Rose Rennekamp, Richard Sawyer, Cynthia Schmeiser, 
and Diane Schnelker. Braden Rood, Jacqueline Snider, and Andrew Welch 
provided assistance in manuscript preparation and bibliographic review. 
Gregory Carrier and Michael Rasmusson provided the graphic design, and 
Ken Kekke was the editorial manager for the report. 

We are grateful for the assistance and support of the aforementioned 
individuals but accept sole responsibility for any errors of omission 
or commission. 

Richard J. Noeth 
Boris B. Volkov 
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Executive Summary 



It is both reasonable and expected that technology should help lead the way 
to improve teaching and learning in our schools. Further, it is reasonable to 
believe that the ability to incorporate the educational opportunities that 
technology promises will help level the playing field throughout K-12 
education — particularly across racial, gender, and geographic divides. 

This policy report provides a view of the issues concerning the effectiveness of 
technology in its role to enhance education. This report is intended for use by 
educational leaders and policymakers who are concerned with making optimal 
use of technology in the schools. Specifically, this report: 

■ Focuses on issues that need to be considered as we assess the impact of 
technology and develop evidence-based strategies for technology integration 
that contribute to high achievement for all students. 

■ Provides useful information and specific recommendations about evaluating 
the effectiveness of technological applications implemented to enhance 
teaching, learning, and achievement. 

Technology should be a tool to help educators meet the educational needs of 
all children. As such, technologies cannot function as solutions in isolation but 
must be thought of as key ingredients in making it possible for schools to 
address core educational challenges 1 . Technology can serve as an enabler in 
teaching and learning to: 

■ Help organize and provide structure for material to students. 

■ Help students, teachers, and parents interact, anytime and anywhere. 

■ Facilitate and assist in the authentication and prioritization of Internet 
material. 

■ Simulate, visualize, and interact with scientific structures, processes, and 
models. 

■ Help in learning history and depicting future trends. 

■ Serve as an extension and enhancer for handicapped populations. 

■ Provide automated translators for multilingual populations 2 . 

However, technology and equity are not inevitable partners. Simply providing 
access does not ensure that technology will effectively enhance teaching and 
learning and result in improved achievement. Nor does providing access imply 
that all teachers and students will make optimal use of the technology. 
Technology may mean little without appropriate objectives and goals for its 
use, structures for its application, trained and skillful deliverers, and clearly 
envisioned plans for evaluating its effectiveness. 



1 Bennett, D., Culp, K. M., Honey, M., Tally, B., & Spielvogel, B. (2000). It all depends: 
Strategies for designing technologies for educational change. Paper presented at the 
International Conference on Learning Technology, Philadelphia, PA. 

2 Bajcsy, R. (2002). Technology and learning. In Visions 2020: Transforming education and 
training through advanced technologies . Washington, DC: U.S. Department of Commerce. 
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Two yardsticks we can use to measure the strides technology has made are 
accessibility by students (and teachers) to technology resources and how 
technology is actually utilized by schools and teachers in different settings and 
for different students. 

The rapid growth of school technology 
infrastructure has led to the increased 
availability and use of computers in schools. 

Most students now have access to computers 
and the Internet in their classrooms, nearly 
all students have access somewhere in their 
schools, and a majority of teachers report 
using computers or the Internet for 
instructional purposes 3 . While the number 
of students per instructional computer 
dropped, those in need are the most likely to 
lack Internet access. However, the ratio is improving rapidly in schools with 
the highest poverty concentration, as has the percentage of instructional rooms 
with Internet access 4 . 

But while they may have abundant computers, schools may not use them in 
the best ways to enhance learning. A great deal depends on the levels of 
planning, structure, preparation, and evaluation of the potential impact that 
technology will have on teaching, learning, and achievement. Experts believe 
that increasing capacity depends on enhancing the technology skills of teachers 
and administrators. Many states, for example, have taken steps to provide 
guidelines for how to use educational technology more effectively; and 80% 
have developed standards for teachers and administrators that include 
technology. 

In terms of utilization, technology has expanded from use primarily as an 
instructional delivery medium to an integral part of the learning environment. 
Technology is serving at least four distinct purposes in the schools: 

■ To teach, drill, and practice using increasingly sophisticated digital content. 

■ To provide simulations and real world experiences to develop cognitive 
thinking and to extend learning. 

■ To provide access to a wealth of information and enhanced communications 
through the Internet and other related information technologies. 

■ As a productivity tool employing application software such as spreadsheets, 
databases, and word processors to manage information, solve problems, and 
produce sophisticated products 3 . 




3 Education Week. (2003, May 8). Technology counts 2003: Pencils down — Technology’s answer 
to testing. 

4 Market Data Retrieval. (2002). Technology in education 2002: A comprehensive report on the 
state of technology in the K-12 market. Shelton, CT: Market Data Retrieval. 

3 Fouts, T. T. (2000). Research on computers and education: Past, present, and future. A report 
to the Rill and Melinda Gates Foundation. Seattle: Seattle Pacific University. 



There seems to be universal agreement that a major criterion of technological 
implementation in the schools should be whether such applications actually do 
improve teaching and learning and increase student achievement. However, 
there also seems to be consensus about the complexity and challenge of 
reliably evaluating the effectiveness of technology. 




It is a daunting task to separate the effects of 
technology from the effects of other factors 
that influence teaching and learning. Results 
and conclusions must be considered in the 
context of the interdependent set of 
variables in which the use of technology is 
embedded. These variables can include 
access, teacher preparation and experience, 
student background, curriculum content, 
instructional methods, and additional 
educational resources. Ascertaining 
technology’s context as well as its impact 
calls for comprehensive evaluations that consist of both formative and 
summative components. Formative evaluations (during the course of the 
program) track the implementation of the technology. Summative evaluations 
(at the end of the program) examine the impact of the technology application. 



Despite schools flooded with computers, the evidence is mixed as to whether 
overall student achievement has notably increased or the achievement gap has 
visibly narrowed as a result. 



Research reviews have generally concurred that: 

■ When combined with traditional instruction, the use of computers can 
increase student learning in the traditional curriculum and basic skills area. 

■ The integration of computers with traditional instruction produces higher 
academic achievement in a variety of subject areas than does traditional 
instruction alone. 

■ Students learn more quickly and with greater retention when learning with 
the aid of computers. 

■ Students like learning with computers and their attitudes toward learning 
and school are positively affected by computer use. 

■ The use of computers appears most promising for low achieving and 
at-risk students. 

■ Effective and adequate teacher training is an integral element of successful 
learning programs based or assisted by technology 6 . 



Vlll 



6 Ibid. 



A great deal of the responsibility for successful integration of technology 
inevitably falls upon individual administrators and teachers. The most critical 
element in technology use is the preparedness and skill level of those who 
employ it. Teachers, for example, need high-quality professional development 
that leads to a professional community centered around the integration of 
technology into the curriculum. Viewed in terms of teaching, many 
recommend that teachers should have basic technology skills and be able to: 

■ Use technology for personal productivity. 

■ Use technology to support learning in a subject area. 

■ Design or adapt technology-supported learning activities. 

■ Manage student-centered, technology-supported activities. 

■ Assess student skills within the context of technology-supported activities 7 . 

Evaluation must pay careful attention to local program contexts. Evaluators must 
first examine the program s specific design describing how interventions are 
expected to bring about particular changes in teaching and learning. Evaluators 
must consider a range of factors such as scope of the evaluation, who the 
stakeholders and partners are, the kinds of data that are needed, and how the 
data will be used. There is often no consistent set of specific curriculum-related 
goals and objectives for the use of technology in teaching and learning. 

Evaluation can consider human and technology inputs (student, teacher, 
school, classroom, and other contextual influences), process (types and areas 
of technology use in school and classroom), and expected and unexpected 
outcomes (student, teacher, family, school, and community achievements). 
Evaluation should not only be limited to outcomes and effects (summative), 
but should also be related to the process of implementing the program, its 
rationale, and the quality of its goals and objectives (formative). 

Evaluation, then, can be formative and summative and include short-term, 
long-term, qualitative, quantitative, conventional, and innovative elements — 
or any combination. It can include a range of procedures — performance 
assessments, standardized tests, observations, writing samples, and other 
indicators of the impact of technology on achievement. Teachers can also 
participate in surveys and focus groups with students (and parents) about use 
of technology in the classroom. 

Key stakeholders play an active role in supporting and modifying the 
evaluation process. The strategic approach to evaluating information systems 
in business, often called critical success factors, may be of use for evaluating 
K-12 technology use. This could involve district and school administrators, 
teachers, parents, and perhaps students convening to determine which factors 
are critical to the success of the technology implementation. 



7 Means, B. (2000). Accountability in preparing teachers to use technology. In Council of Chief 
State School Officers, 2000 State Educational Technology Conference Papers. Washington, DC: 
Council of Chief State School Officers. 
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The issues involved in evaluating the effectiveness of technology in education 
are complex. Yet technology, as a primary educational tool and major school 
expenditure, must be held accountable to its promise of enhancing teaching, 
learning, and achievement. Each day, educational leaders and policymakers 
at all levels are faced with questions and decisions about technology. Reliable 
information to help answer the questions and guide these decisions comes 
from comprehensive planning with key technology stakeholders and sound 
evaluation plans and practices. Following are a series of questions to address 
when considering technology evaluation: 

■ How and when will evaluation of technology’s impact on teaching, learning, 
and achievement be done? 

■ Who will be responsible for collecting ongoing data to assess 
technology’s effectiveness? 

■ How will accountability for implementation be assessed? 

■ How will the level of technological proficiency of students and 
teachers be assessed? 

■ How will technology be used to evaluate teaching and learning? 

■ What is the key indicator of success for each component of the 
technology plan? 

■ How will the effectiveness of disbursement decisions in light of priorities 
be analyzed? 

■ How will implementation decisions to accommodate for changes as a result 
of new information and technologies be analyzed 8 ? 

Given the critical need to effectively evaluate the teaching, learning, and 
achievement outcomes of technology, we offer three recommendations that 
can serve as reasonable benchmarks for those faced with the challenges of 
assessing the accountability of their school’s and district’s applications of 
technology: 

1. All relevant stakeholders should reach consensus on the purpose and 
intended outcomes of the planned technology implementation . 

2. Every technology plan should include an evaluation component, and 
multiple evaluation methods should be considered, specified, and 
employed to assess agreed-upon outcomes. 

3. Administrators and teachers should receive adequate, tailored, and 
continuing education about how to best integrate technology into 
their schools and courses, and should be evaluated on their 
proficiency in doing so. 



x 



8 North Central Regional Education Laboratory. (2003). Evaluating the implementation of your 
technology plan. Oak: Brook, IL: North Central Regional Educational Laboratory. 
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Introduction 



Technology is evolving at an astonishing rate. It has dramatically changed the 
ways we work, learn, interact, and spend our leisure time. Computers and 
information technologies have visibly revolutionized nearly every aspect of 
daily life — how and where we get our news, how we order goods and services, 
and how we communicate. 

It is both reasonable and expected that technology should also help lead the 
way to improve teaching and learning in our schools. Further, it is reasonable 
to believe that the ability to incorporate the educational opportunities that 
technology promises will help level the playing field throughout K-12 
education — particularly across racial, gender, and geographic divides. 

Technology offers new ways of teaching and learning, and provides new ways 
for all involved in education to be openly accountable to parents, communities, 
and students (National Research Council, 1995). The National Academy of 
Sciences suggests that new and emerging technologies have the potential to 
enhance learning and the development of new knowledge in many exciting 
ways by providing access to a vast array of information and connections to 
other people — for information, feedback, and inspiration (National Research 
Council, 1999). 

Despite such promise, however, there is varied evidence regarding the 
effectiveness of technology as it relates to educational equity and achievement 
issues (Barton, 2001; CEO Forum on Education & Technology, 2001; Cuban, 
2001). One of the key goals of the No Child Left Behind Act is to enhance 
education through technology, with a specific focus on what works in 
technological applications (U.S. Department of Education, 2002a). Relatedly, 
the revised National Technology Plan (U.S. Department of Education, 2000) 
specifically calls for empirical studies to be conducted in schools and 
classrooms that are designed to determine which uses of technology are most 
effective — under which conditions and with which students. 

This policy report provides a view of the issues concerning the effectiveness of 
technology in its role to enhance education. Partially in response to the 
technology focus of No Child Left Behind, this report examines much of the 
information available on the evaluation of technology in fostering educational 
equity and achievement. This report is intended for use by educational leaders 
and policymakers who are concerned with making optimal use of technology in 
the schools. Specifically, this report: 

■ Focuses on issues that need to be considered as we assess the impact of 
technology and develop evidence-based strategies for technology integration 
that contribute to high achievement for all students. 

■ Provides useful information and specific recommendations about evaluating 
the effectiveness of technological applications implemented to enhance 
teaching, learning, and achievement. 



1 



2 

Technology in Our Schools 



There is widespread belief that thoughtful and pervasive applications of 
technology 1 can help eliminate many educational inequities between inner city 
and suburban schools, between urban and rural districts, and those affecting 
minorities and people with physical and learning disabilities. Technology 
should be a tool to help educators meet the educational needs of all children. 
As such, technologies cannot function as solutions in isolation, but must be 
thought of as key ingredients in making it possible for schools to address core 
educational challenges (Bennett, Culp, Honey, Tally, & Spielvogel, 2000). 

Bajcsy (2002) views technology in teaching and learning as an enabler and 
suggests that technology can work to: 

■ Help organize and provide structure for material to students. 

■ Help students, teachers, and parents interact, anytime and anywhere. 

■ Facilitate and assist in the authentication and prioritization of 
Internet material. 

■ Simulate, visualize, and interact with scientific structures, processes, 
and models. 

■ Help in learning history and depicting future trends. 

■ Serve as an extension and enhancer for handicapped populations. 

■ Provide automated translators for multilingual populations. 

Similarly, Wilson (2002) envisions technology as offering endless possibilities 
to enhance educational experiences, expand academic opportunities, and 
develop critical employment skills. Others affirm the real promise of 
technology to be in its potential to facilitate fundamental, qualitative changes 
in the nature of teaching and learning (Thompson, Schmidt, & Stewart, 2000). 

However, technology and equity are not inevitable partners (Johnson, 2002; 
Resnick, 2002; Whitehead, Jensen, & Boschee, 2003). Simply providing access 
does not ensure that technology will effectively enhance teaching and learning, 
and result in improved achievement. Nor does providing access imply that all 
teachers and students will make optimal use of the technology. Technology 
may mean little without appropriate objectives and goals for its use, structures 
for its application, trained and skillful deliverers, and clearly envisioned plans 
for evaluating its effectiveness. 



1 The term “technology” often refers to a wide range of computer-based teaching and learning 
materials and applications, including all elements of computer use, Internet resources, various 
electronic communications, and distance education. Thus the terms technology and computers 
are used interchangeably in this report. 
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The Benton Foundation Communications Policy Program (2002) suggests that 
five factors must be in place for technologies to support real gains in 
educational outcomes: 

■ Leadership around technology use, anchored in solid educational objectives. 

■ Sustained and intensive professional development that takes place in the 
service of the core vision, not simply around technology. 

■ Adequate technology resources in the schools. 

■ Recognition that real change and lasting results take time. 

■ Evaluation that enables school leaders and teachers to determine whether 
they are realizing their goals and to help them adjust their practice to better 
meet those goals. 

Similarly, two primary elements of the Technology Standards for School 
Administrators are leadership and vision (Technology Standards for School 
Administrators Collaborative, 2001). Under these standards, educational 
leaders are called upon to: 

■ Facilitate shared development by all stakeholders of a vision for technology 
use and widely communicate that vision. 

■ Maintain an inclusive and cohesive process to develop, implement, and 
monitor a dynamic, long-range, and systematic technology plan to achieve 
the vision. 

■ Advocate for research-based effective practices in the use of technology. 

Accessibility and Utilization 

Two yardsticks for measuring the strides 
technology has made are accessibility by 
students (and teachers) to technology 
resources and how technology is actually 
utilized by schools and teachers in different 
settings and for different students. 

Regarding accessibility, it is possible that 
more has been invested in the effort to 
incorporate computer technology into the 
K-12 system than in any other educational 
advance in history (Poole, 2001). 

Computer Access. The rapid growth of school technology infrastructure has 
led to the increased availability and use of computers in schools. Most students 
now have access to computers and the Internet in their classrooms, nearly all 
students have access somewhere in their schools, and a majority of teachers 
report using computers or the Internet for instructional purposes (Education 
Week, 2003). According to the Department of Education (2002b), school 
Internet connectivity has grown from 35% in 1994 to 99% in 2001, while 
classroom connectivity — what counts most for instructional purposes — has 
increased from 3% in 1994 to 87% in 2001. 
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According to Market Data Retrieval (2002), the number of students per 
instructional computer dropped between 1984 and 2002 from 125 to 3.8 (or 
5.6 per Internet-connected computer), suggesting that as more computers are 
added to classrooms, fewer students have to share them. At the same time, 
students in need are the most likely to lack Internet access. In 2001, 
high-poverty schools had 6.8 students per instructional computer with Internet 
access, compared to 4.9 students per computer in low-poverty schools. 
However, the ratio has improved rapidly (from 9.1 students in 2000) in schools 
with the highest poverty concentration. Similarly, the percentage of 
instructional rooms with Internet access in these schools increased between 
2000 and 2001, from 60% to 79% in schools with the highest concentration of 
poverty and from 64% to 81% in schools with the highest minority enrollment 
(U. S. Department of Education, 2002b). 

Extended access in schools has increased as well. For students without home 
Internet access, many schools provide such access outside of regular hours 
(e.g., before and after school). The U. S. Department of Education (2002b) 
reports that 51% of schools with Internet access made computers available 
to students outside of regular school hours. In addition, more than 
20,000 technology services are spread across the nation and provide 
everything from specialized computer training to low-interest loans to help 
financially challenged families purchase personal computers for use at home 
(Education Week, 2001). These venues include public libraries, Department 
of Housing and Urban Development Neighborhood Network Sites, and 
Urban League centers. 

New technology applications also are happening at an incredible pace. 

The Southern Regional Education Board (SREB, 2002a) reports that virtual 
learning is coming to K-12 schools faster than most realize. During the 
2001-02 school year, SREB estimated that nationwide over 50,000 middle and 
high school students were enrolled in online courses. 

Computer Use. But while they may have abundant computers, schools may 
not use them in the best ways to enhance learning. Dede (2002) reminds us 
that the important issue in effectiveness for learning is not the sophistication 
of the technologies, but the ways in which their capabilities aid and motivate 
users. The existence of a particular technology does not dictate the manner in 
which it will be used. Viewed in terms of teaching, Means (2000) recommends 
that teachers should have basic technology skills and be able to: 

■ Use technology for personal productivity. 

■ Use technology to support learning in a subject area. 

■ Design or adapt technology-supported learning activities. 

■ Manage student-centered, technology-supported activities. 

■ Assess student skills within the context of technology-supported activities. 
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A great deal depends on the levels of planning, structure, preparation, and 
evaluation of the potential impact that technology will have on teaching, 
learning, and achievement. Experts believe that increasing capacity depends 
on enhancing the technology skills of teachers and administrators — which is 
supported by the No Child Left Behind Act. Many states, for example, have 
taken steps to provide guidelines for how to use educational technology more 
effectively; and 80% have developed standards for teachers and administrators 
that include technology (Education Week, 2003). 

In terms of utilization, Fouts (2000) reports that in the past decade technology 
has expanded from use primarily as an instructional delivery medium to an 
integral part of the learning environment. He indicates that technology is 
serving at least four distinct purposes in the schools: 

■ To teach, drill, and practice using increasingly sophisticated digital content. 

■ To provide simulations and real world experiences to develop cognitive 
thinking and to extend learning. 

■ To provide access to a wealth of 
information and enhanced 
communications through the Internet and 
other related information technologies. 

■ As a productivity tool employing 
application software such as spreadsheets, 
databases, and word processors, to 
manage information, to solve problems, 
and to produce sophisticated products. 

How technology is actually utilized is a 
critical issue, and many schools may be using 
computers in ways that may not contribute significantly to their entire 
population s productive learning. Different groups of students within schools 
may use the computer in different ways. Female students have often steered 
(and been steered) away from science and technology options and have 
traditionally tended to use computers for more basic (e.g., word processing) 
rather than more complex (e.g., scientific modeling) purposes (Creighton, 
2003; Education Week, 2001). Bolt and Crawford (2000) report a technology 
gender gap, much like in other aspects of society, where females tend to view 
computer technology with specific uses in mind rather than explore its myriad 
opportunities and applications. 

Teachers of poor or minority children have often had a propensity to assign 
remedial drill computer programs rather than those demanding higher-order 
thinking and mastery of challenging concepts, including use of the computer 
for Internet research (National Alliance of Business, 2002). African American 
and Hispanic students have lagged behind in access to tasks involving 
simulations and applications that exercise higher-order thinking skills as 
opposed to drill and practice (Yau, 1999). Many teachers in high-poverty 
schools are less likely to have training both in technology-enhanced curriculum 
development and in using computers in their classrooms. They are also less 
likely to engage their students in solving complex problems using computers. 
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Teachers can have different expectations for how technology can be used for 
high and low achievers that may widen the achievement gap rather than 
narrow it. Education Week (2001) showed that teachers often believe that 
students with learning difficulties are less likely than their higher-achieving 
classmates to have rewarding experiences with technology. Moreover, devising 
methods for using challenging software with lower-achieving students takes 
considerable time, training, and practice. Schools with the highest poverty and 
minority concentrations were less likely to have special hardware and software 
for students with learning and physical disabilities than were schools with 
lower concentrations (U. S. Department of Education, 2002b). 

Teachers of bilingual students may limit the use of technology in their classes 
as well. There is often little use by English-as-second-language students of 
computer labs and applications. In addition, there may not be a bilingual 
teacher or aide to assist in the lab (Education Week, 2001). Bilingual and 
English-as-a-second-language programs are also less likely to be given 
computers than mainstream programs or may receive obsolete machines with 
dated drill and practice software (Creighton, 2003). 

Providing equal access to technology may not signify equal educational 
opportunity nor reduce the achievement gap for disadvantaged students. 
Children with disabilities, or who are minority, poor, or low achievers, may be 
left behind after the introduction of computers into schools. Even high-tech 
computers may often become not much more than trivial workbooks and 
control mechanisms for students in schools with predominantly minority 
enrollments (Education Week, 2001). 
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3 

Evaluating Effectiveness of Technology as an 
Instructional Tool 

There seems to be universal agreement that a major aspect of technological 
implementation in the schools should be whether such applications actually do 
improve teaching and learning and increase student achievement (Southern 
Regional Education Board, 2002b; U. S. Department of Education, 1998; 

U. S. Department of Education, 2002c). However, there also seems to be 
universal agreement about the complexities and challenges of reliably 
evaluating the effectiveness of technology (North Central Regional Education 
Laboratory, 1999; U. S Department of Education, 2002c; WestEd, 2002). 

Technology is an integral part of our educational system, and it is a daunting 
task to separate the effects of technology from the effects of other factors that 
influence teaching and learning (Thompson, Schmidt, Walker, O’Connell, 
Bergland, Bengfort, & Linduska, 2000). Results and conclusions must be 
considered in the context of the interdependent set of variables in which the 
use of technology is embedded (Collis & Lai, 1996; Owen, Calnin, & Lambert, 
2002; Russell, 2001). 

These variables can include access, teacher preparation and experience, 
student background, curriculum content, instructional methods, and additional 
educational resources. Understanding technology’s context as well as its impact 
calls for comprehensive evaluations that consist of both formative and 
summative components. Formative evaluations (during the course of the 
program) track the implementation of the technology. Summative evaluations 
(at the end of the program) examine the impact of the technology application. 

Some of the Evidence 

Despite schools flooded with computers, the evidence is mixed as to whether 
overall student achievement has notably increased or the achievement gap has 
visibly narrowed as a result of the use of technology (Barton, 2001; Cuban, 
2001; Healy, 1998; Wenglinsky, 1998; Wilson, 1999; Yau, 1999). The following 
section presents selected reports that describe the differential and categorical 
effects of technological applications on teaching, learning, and achievement. 
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Summarizing the reviews of research on computers and education, Fouts (2000) 
indicates that, while not all reviews show outcomes in favor of computer use, the 
vast majority reach positive conclusions about their efficacy. He reports general 
concurrence that: 

■ When combined with traditional instruction, the use of computers can 
increase student learning in the traditional curriculum and basic skills areas. 

■ The integration of computers with traditional instruction produces higher 
academic achievement in a variety of subject areas than does traditional 
instruction alone. 

■ Students learn more quickly and with greater retention when learning with 
the aid of computers. 

■ Students like learning with computers and their attitudes toward learning 
and school are positively affected by computer use. 

■ The use of computers appears most promising for low achieving and 
at-risk students. 

■ Effective and adequate teacher training is an integral element of successful 
learning programs based on or assisted by technology. 

He cautions that much of the research has been criticized for its low quality 
and suggests that these results are not guaranteed by the simple introduction 
of computers and related technology into the classrooms. Many other factors 
play important roles in the process. 




WestEd (2002) examined selected research studies, ones they judged to be the 
most methodologically sound and that analyzed change over time. They 
concluded that a number of studies have provided convincing evidence that 

technology can be effective in teaching basic 
skills, and that computer-assisted instruction 
and drill and practice software can 
significantly improve scores on standardized 
achievement tests. They also state that 
technology can provide the means for 
students with special needs to communicate 
via e-mail and use the Internet for research, 
and also help teachers accommodate 
students’ varying learning styles. They believe 
that there is substantial research that 
suggests technology can have a positive effect 
on student achievement under certain circumstances and when used for certain 
purposes. They list a number of key conditions that have repeatedly appeared 
in the literature as crucial elements for successfully using technology: 




■ Technology is best used as one component in a broad-based reform effort. 



■ Teachers must be adequately trained to use technology. 



■ Technological resources must be sufficient and accessible. 

■ Effective technology use requires long-term planning and support. 



■ Technology should be integrated into the curricular and instructional 
framework. 



8 



The Benton Foundation Communications Policy Program (2002) concludes 
that after more than two decades of research on the benefits of technology, 
evidence that demonstrates its positive effects on achievement is mounting. 
Specifically they conclude that: 

■ Large-scale statewide implementation of technology correlates with 
increased performance on standardized tests. 

■ Software that supports the acquisition of early literacy skills — including 
phonetic awareness, vocabulary development, reading comprehension, and 
spelling — can support student learning gains. 

■ Mathematics software, particularly programs that promote experimentation 
and problem solving, enables students to embrace key mathematical 
concepts that are otherwise difficult to grasp. 

■ Scientific simulations, microcomputer-based laboratories, and scientific 
visualization tools have all been shown to result in students’ increased 
understanding of core science concepts. 

They caution that, if technologies are used to support educational outcomes, 
there must be sustained professional development, technology leadership 
anchored in solid educational objectives, and evaluation that will help 
determine whether educational goals are realized and whether educational 
practices are appropriate. 

In a report commissioned by the Software and Information Industry 
Association, Silvin-Kachala and Bialo (2000) summarized educational 
technology research from the late 1980s through 2000. They concluded that 
technology is making a significant positive impact on education. In addition to 
many of the preceding findings, they listed the following: 

■ A learning advantage has been found when students have developed 
multimedia presentations on social studies topics. 

■ Kindergartners who have used technology have benefited in areas such as 
improved conceptual knowledge, reading vocabulary, reading 
comprehension, and creativity. 

■ Educational technology has significant positive effects on student attitudes 
and achievement for special needs population — speech recognition being a 
valuable compensatory tool for the learning disabled. 

■ Use of online telecommunications for collaboration across classrooms in 
different geographic locations can improve academic skills. 

■ Technology has been found to have positive effects on student attitudes 
toward learning and on student self-concept. 

■ Students trained in collaborative learning on computers in small groups had 
higher achievement, higher self-esteem, and better attitudes toward 
learning, and these results were especially pronounced for low ability and 
female students. 

They concluded that the specific student population, the software design, the 
educator s role, how the students are grouped, the preparedness of the 
educator, and the level of student access to the technology all influence the 
level of effectiveness of educational technology. 
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Finally, a study was recently requested by the Canadian Education Statistics 
Council to examine the impact of technology on that country’s education. 
Among the results (Ungerleider & Burns, 2002) were that student attitudes 
toward computers and computer-related technologies improved as a 
consequence of exposure; the use of technology for group work was beneficial 
if teachers took into account the interplay among age of the students, kind of 
task, and amount of independence allowed; and the use of technology for 
mathematics instruction has a significantly positive effect on teaching high 
level concepts to students in grade eight and above. 

Considerations for Evaluation 

Dominant themes in the preceding reviews have implications for the design 
and implementation of technology evaluations. A great deal of the 
responsibility for successful integration of technology inevitably falls upon 
individual administrators and teachers. The most critical element in 
technology use is the preparedness and skill level of those who employ it 
(Wilson, 2002). Hart, Allensworth, Lauen, and Gladden (2002) suggest that 
once administrators provide students and teachers with sufficient and reliable 
technology, essential supports are needed to propel its use forward. Teachers, 
for example, need high-quality professional development that leads to a 
professional community centered around the integration of technology into 
the curriculum. 

Education Week (2003) reports that some states have adopted technology 
requirements for initial licensure for teachers and/or administrators and 
several states require technology training or coursework for teacher or 
administrator recertification. Ten states currently offer professional or financial 
incentives for teachers to use educational technology, and 31 states provide 
such incentives for administrators. 

Student achievement is often mediated by the processes teachers use to 
integrate technology into instruction. Technology can be used for drill and 
practice; it can be used to promote critical, analytic, and higher-order thinking 
skills, as well as real-world problem solving. Technology in schools may best 
be used in the ways adults use technology to accomplish their work — write, 
organize and analyze information, do research, and communicate 
(Rockman, 2000). The ability of teachers to foster such changes depends upon 
training that shows them how to integrate technology into content-specific 
instructional methods (Heinecke, Blasi, Milman, & Washington, 1999). 

Cuban (2001) emphasizes a systematic approach to evaluating technological 
implementations that involves: 

■ Identification of educational needs. 

■ Specification of implementation goals. 

■ Design of instructional strategies to create effective learning environments. 

While the conclusions about the efficacy of technology on achievement are 
mixed, some believe that the fault lies not with the technology-based 
innovations, but rather with evaluation plans and tools. Only a small number of 
school districts have established guidelines for evaluating the impact of 
educational technology (Whitehead et al., 2003). 




Penuel and Means (1999) recommend that evaluation must pay careful 
attention to local program contexts. Evaluators must first examine the 
program’s specific design describing how interventions are expected to bring 
about particular changes in teaching and learning. They advise that anticipated 
changes must be described in enough detail for stakeholders to know when 
desired changes have been achieved. 

Second, evaluators must consider a range of 
factors such as scope of the evaluation, who 
the stakeholders and partners are, the kinds 
of data that are needed, and how the data 
will be used. There is often no consistent set 
of specific curriculum-related goals and 
objectives for the use of technology in 
teaching and learning. 

The Iowa Consortium for Assessment of 
Learning with Technology (Thompson, 

Schmidt, Walker, O’Connell, Bergland, 

Bengfort, & Linduska, 2000) suggests that there are three fundamental 
elements that create the foundation for evaluation: school improvement, 
research, and technology. The Consortium explains that specific curriculum 
standards are selected as targets to inform teachers’ work with technology, 
technology represents the cognitive tools that students use to impact and 
change the way they learn, and research will help design studies that 
contribute to the existing knowledge base. 



Given the changing demographics of our society, it is important to know how 
students from different backgrounds acquire content meaning and come to 
new understandings through the use of computers. Social interactions in the 
classroom and the social aspects of school culture also affect the impact of 
educational technology (Zhao, Byers, Pugh, & Sheldon, 2001). School, 
community, and family culture are principal elements of the student’s system 
of influences; this culture shapes the system and the subsequent outcomes of 
technology use (Peled, Peled, & Alexander, 1996). Evaluators should seek to 
understand the features of the technology implementation and its potential 
impact upon the social and ecological composition of the classroom. 



Evaluation should grasp the effects of using technology at individual, 
organizational, and sometimes even community levels. This type of evaluation 
may be based on a system of learning benchmarks and other new means of 
assessments that take the context of evaluation into account (McNabb, 

Hawkes, & Rouk, 1999). The evaluation should depend on the educational 
needs, goals, setting, technology application, and expected outcomes 
(Milone, 1996; Russell, 2001). Rossi, Freeman, and Lipsey (1999) suggest that 
every evaluation be tailored to the particular purposes and circumstances of a 
given program so that it will be capable of yielding credible and useful answers 
to the specific questions at issue while still being sufficiently practical to 
actually implement with available resources. 
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As a rule, the closer the outcome measure is to actual student performance, the 
more confident evaluators can be about the impact of the educational program, 
including technological applications, on that performance (Kennedy, 1999). 
Depending on the educational needs and goals of the program, which are often 
tied to specific programs or funded projects, outcome measures might include 
changes in disciplinary referrals or completed homework assignments. In the 
long term, changes in test scores and other measures of performance, increased 
college attendance, increased job offers, measures of higher-order thinking 
skills, more sophisticated communication skills, research skills, and social skills 

might be included. Other measures might 
be perceptions about implementation 
benefits, attitudes toward learning, 
motivation, self-esteem, engagement levels, 
and retention (Fouts, 2000; Heinecke et al., 
1999; Silvin-Kachala & Bialo, 2000; 
Ungerleider & Burns, 2002). 

Ultimately, evaluation plans should reflect 
beliefs about how technology fits into the 
model of instruction, how it is utilized to 
improve teaching and learning, and, in so 
doing, how it increases student achievement, 
motivation, and value for learning (Russell, 2001). In addition, evaluation plans 
can include efforts to determine enhanced efficiency and cost-effectiveness, 
and examine whether technology is achieving better results at lower costs. 
Adequate evaluation can help improve existing technology programs so that 
they evolve to meet student and teacher needs, work to support multiple levels 
of learning, and foresee new developments. 

Approaches to Evaluation. Evaluation can consider human and technology 
inputs (student, teacher, school, classroom, and other contextual influences), 
process (types and areas of technology use in school and classroom), and 
expected and unexpected outcomes (student, teacher, family, school, and 
community achievements). Evaluation should not be limited only to outcomes 
and effects (summative), but should also be related to the process of 
implementing the program, its rationale, and the quality of its goals and 
objectives (formative) (McNabb et al., 1999). 




Evaluation, then, can be formative and summative and include short-term, 
long-term, qualitative, quantitative, conventional, and innovative elements — 
or any combination. It can include a range of procedures — performance 
assessments, standardized tests, observations, writing samples, and other 
indicators of the impact of technology on achievement. Teachers can also 
participate in surveys and focus groups with students (and parents) about 
technology use in the classroom. 

Key stakeholders play an active role in supporting and modifying the 
evaluation process. The strategic approach to evaluating information systems 
in business — often called critical success factors (Rockart, 1999) — may be 
of use for evaluating K-12 technology use. This could involve district and 
school administrators, teachers, parents, and perhaps students convening 
to determine which factors are critical to the success of the technology 
implementation. They would then rank them according to their significance 
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and determine the role that technology should play in supporting the decision- 
makers in dealing with efficacy issues. This method provides a focus on issues 
that stakeholders regard as important and reflects a Baldrige quality approach 
to technology implementation (Walpole & Noeth, 2002). Bennett et al. (2000) 
provide a number of useful characteristics related to effectively evaluating 
technology. These include: 

■ Assumptions 

> Technologies in and of themselves rarely bring about substantial change 
in teaching and learning. 

> The impact of technology on specific aspects of teaching and learning 
can be usefully understood only in context. 

■ Methodological features 

> Evaluation is largely process oriented. 

> It is oriented toward change rather than doing better within the old 
framework. 

> Evaluation is multidisciplinary, combining elements from different 
fields. 

■ Design elements 

> Long-term collaborations with teachers. 

> Systemic integration and research on the impact of innovations across 
multiple levels of the school system. 

Building the evaluation capacity of districts and schools is critical, as is the 
willingness of staff to evaluate their efforts. Teachers, technology coordinators, 
school administrators, and evaluators can collaborate to identify the impacts 
associated with technology uses. Evaluation should build the capacity of teachers 
to assess technology resources and alignment of their uses with learning goals 
and content standards. McNabb et al. (1999) explain that some of the best 
results in evaluating technology come from schools that recognize and harness 
the expertise teachers have in identifying technology-induced learning 
outcomes. Having teachers train teachers to evaluate the effects of technology in 
the classroom presents a powerful professional development strategy. 

Judgments about the effectiveness of technology should be made on the basis 
of specific, documented criteria. These criteria can be formulated at the outset 
of the implementation after gaining a thorough understanding of the nature of 
the implementation and the perceptions of its stakeholders. To diagnose 
strengths and weaknesses effectively, administrators and policy makers must 
know what outcomes are expected. 

Educational programs and educational technology plans should have written 
objectives that provide understandable levels of detail. Where objectives are 
vague, evaluators should work closely with administrators to understand and 
clarify what the target audience should be able to know and do after 
technology implementation. Evaluation information will be especially useful 
when target goals and outcomes are agreed upon and documented by 
stakeholders and evaluators. Evaluators should also be attentive to the 
possibility of unplanned effects that might contribute to or hinder 
achievement. 



4 

Recommendations 



The issues involved in evaluating the effectiveness of technology in education 
are complex. Yet technology as a primary educational tool and major school 
expenditure must be held accountable to its promise of enhancing teaching, 
learning, and achievement. While information is accruing, available evidence 
indicates that technology generally has demonstrated positive but limited 
results on improving the educational achievement of all students. Clearly, 
more information of a more rigorous nature is needed. 

Each day, educational leaders and policymakers at all levels are faced with 
questions and decisions about technology. Reliable information to help answer 
the questions and guide these decisions comes from comprehensive planning 
with key technology stakeholders and sound evaluation plans and practices. To 
this end, the North Central Regional Educational Laboratory (2003) offers a 
series of questions to address when considering technology evaluation: 

■ How and when will evaluation of technology’s impact on teaching, learning, 
and achievement be done? 

■ Who will be responsible for collecting ongoing data to assess 
technology’s effectiveness? 

■ How will accountability for implementation be assessed? 

■ How will the level of technological proficiency of students and teachers 
be assessed? 

■ How will technology be used to evaluate teaching and learning? 

■ What is the key indicator of success for each component of the 
technology plan? 

■ How will the effectiveness of disbursement decisions in light of priorities 
be analyzed? 

■ How will implementation decisions to accommodate for changes as a result 
of new information and technologies be analyzed? 

There are numerous resources to support the gathering of such information, 
from both the public and private sectors. A number of resources are listed in 
the bibliography of this report, and two federal resources that cover the 
necessary framework and details of technology evaluation in the schools are 
particularly worth noting: 

■ An Educators Guide to Evaluating the Use of Technology in Schools and 
Classrooms (U.S. Department of Education, 1998) 

■ Technology in Schools: Suggestions , Tools , and Guidelines for Assessing 
Technology in Elementary and Secondary Education (U. S. Department of 
Education, 2002c). 
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Given the critical need to effectively evaluate the teaching, learning, and 
achievement outcomes of technology, we offer three recommendations that 
educational leaders and policymakers can include as part of all technology 
planning and evaluation. Though not exhaustive, they can serve as reasonable 
benchmarks for those faced with the challenges of assessing the accountability 
of their schools and district s applications of technology. 



1. All relevant stakeholders should reach consensus on the purpose and 
intended outcomes of the planned technology implementation. 

The use and impact of technology in the school setting is far-reaching. There 
are many stakeholders who will influence or be influenced by technological 
implementations in the schools. Technology affects not only those most 
directly involved in teaching and learning (teachers and students), but also 
other stakeholders: administrators, other educators and school staff, parents, 
teachers and learners not directly involved in the implementation, community 
members, business leaders, school board 
members, and legislators. As districts and 
schools consider and reconsider technology 
implementations, they should include 
representatives of all groups who might 
potentially be affected, both in the short and 
long term, by the technology application(s). 

The purpose and expected outcomes should 
be part of the discussions that conceptualize 
and create the technology implementation. 

The net that considers potential outcomes 
should be cast widely — to include not only 

the individuals and groups who might be impacted but also the content and 
context areas that might be affected (subject matter learning, social 
interaction, collaborative learning, tangential learning areas). All stakeholders 
should strive to reach consensus on the purpose and intended outcomes of the 
technology. This consensus should be documented as specifically as possible, 
including the qualification and quantification of all anticipated outcomes. 




2. Every technology plan should include an evaluation component, and 
multiple evaluation methods should be considered, specified, and 
employed to assess agreed-upon outcomes. 

Schools and districts should have manageable technology plans that include a 
major focus on evaluation activities and outcomes — keeping the perspective 
that technological applications are one element within a complete instructional 
process. The discussion and design of the evaluation component should begin 
when technology programs are conceptualized and continue throughout (and 
beyond) program implementation. The evaluative element in technology 
programs should not be considered an add-on after the fact. Its components 
(e.g., goals, activities, measures, indicators, benchmarks, reporting methods) 
should be developed and agreed on by stakeholders. (Even if a program has 
begun without an evaluation component, it still may not be too late to institute 
an evaluation!) 



Considerations should include the issues of formative evaluation (conducted 
during the implementation, allowing for mid-program refinements) and 
summative evaluation (end of implementation to determine effectiveness) in 
terms of importance to stakeholders, resources required, timing, and expected 
information received. Another consideration might be short-, medium-, and/or 
long-term evaluations — when they should be undertaken, how they would fit 
together, and what information might be gathered from each. The next level 
of consideration might include the types of evaluation (qualitative and 
quantitative) that might be conducted at each stage — surveys, standardized 
assessments, locally-developed measures, focus groups, teacher 
proficiency indexes. 

3. Administrators and teachers should receive adequate, tailored, and 
continuing education about how to best integrate technology into 
their schools and courses and should be evaluated on their 
proficiency in doing so. 

Administrators and teachers are key technological interfaces in the schools: 
one is responsible for bringing technology into the district or building, the 
other for bringing it into the classroom. They are also major technology 
stakeholders. Successful technological implementations will largely depend 
upon the motivation, knowledge, and skill of administrators and teachers to 
implement and utilize technology in effective ways to enhance learning for all 
students. It is imperative that these educators be fully supported in this regard 
through adequate pre-service preparation, ongoing and state-of-the-art 
in-service activities, and links to local colleges and other resources for 
additional support and learning. 

In return, administrators and teachers must be held accountable for the 
effectiveness of their uses of technology to support an enhanced learning 
environment for the educational community, as well as for subject matter 
learning for the range of students found in their classrooms. In other 
workplace environments, as new tools are incorporated into the work cycle 
workers are evaluated on their proficiency in utilizing them. Similarly, as 
educators are taught how to utilize technology to support teaching and 
learning, they should be held accountable for their ability to do so effectively. 
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